Scanner Refactor #3240
Merged
Scanner Refactor #3240
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Optimized GetFileWithCertainExtensions to use yield and reduce memory overhead.
…otChangedSinceLastScan() says nothing has changed when a directory underneath has been deleted/renamed/etc thus missing out on critical work. I need a new strategy to handle this case.
… files. This avoids the issues with OS not reporting good data and us having to over scan. Specials don't group, so I need to look into that.
…cate scans on image libraries (like the last write doesn't work). I noticed some weird parsing with Special directories, so I excluded them a bit. Some subdirectories will always report dirty, usually image libraries with subfolders or weird layout. Not much I can do.
…nga and comicvine type layouts. Code is also much easier to understand.
…1 was already scanned. There is only an issue with image layouts where chapters are being replaced. Likely due to not merging in TrackSeries.
…f chapters due to the new scanning mechanism.
/ <- Lib root assume /love hina/love hina/v01.cbz /love hina/specials/sp01.cbz The lowest series folder was /love hina/love hina/ for some reason, meaning series scan wasn't getting the full series (all folders) scanned in I'm not sure if I'm still on a good track or not with the scanner loop. It works on some stuff and others not well. But I am finding bugs in other pieces of code, so I will continue on this path to hopefully cleanup more of the scanner.
…an parser info by parser info. Due to the scanner processing incomplete changes now, we need to shift it up.
majora2007
added
enhancement
Added a new code that will create all genres ahead of the scan to avoid any FK issues.
…e to yet another bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a canary release to test out the new scanner. These scanner enhancements are part of an effort to bring more reliability into the scanner and have ingestion of changes after the first scan to be faster. This new code is likely to make the first scan take longer, but with the scope of what Kavita does, it's acceptable for a first scan to take some time.
For those that are testing this, please report back to me on scan time for first scan (vs size of library), if ingestion is fast after the fact and if files are being picked up and scanned correctly (ie, updated a nested directory, you should only see that directory scanned).
Do not try to use this for your main server. I make no guarantees that it will work exactly (although from spot checking against my prod library, it's looking pretty good)
Changed
Fixed
Example:
/ <- Lib root assume
/love hina/love hina/v01.cbz
/love hina/specials/sp01.cbz
The lowest series folder was /love hina/love hina/ for some reason, meaning series scan wasn't getting the full series (all folders) scanned in.